Large in-memory databases are currently being used on production bases to serve as key-value stores, online transaction processing (OLTP) databases, large-scale web applications, massive online multi-player games, software defined networks (SDN) controller network views, scientific simulation, etc. These applications are built to sustain high transaction rates using multiple execution contexts, e.g., threads in non-uniform memory access (NUMA) shared-everything architectures, whereby the dataset is logically partitioned and each partition is associated with an execution context which is responsible to process transactions on its partition.
The partitioning strategy enforces the exclusiveness of an execution context upon a partition such that the transaction is executed in a single-thread physical access environment without taking any locks or latches, as described in Kalman, Robert et al.: H-store: A High-performance, Distributed Main Memory Transaction Processing System, VLDB Endowment, August 2008. This approach may be described as combining high efficiency of serial processing on a partition, with parallelism employed across the partitions enabling scaling-up transaction processing.
Static partitioning strategy can, however, entail efficiency problems and severely limit the scalability when even a slightly skewed workload distribution is imposed on the data-set, as some executing contexts may encounter exceptionally high update rate, while the other ones being under-worked, rendering the partitioning useless. As some authors state, the varying workload skew in real world examples is rather the rule, than an exception, as described in Pinar Tozun et al.: Scalable and dynamically balanced shared-everything OLTP with physiological partitioning, The VLDB Journal DOI 1007/s00778-012-0278-6, June 2012.
Hyeontaek Lim et al.: MICA: A Holistic Approach to Fast In-Memory Key-Value Storage, NSDI'14, April 2014, Seattle, Wash. USA describes an in-memory key-value store which addresses the workload skew by utilizing a special static hash function trying to distribute the placement of keys in partitions such that in average the load on a partition is under 53% deviation on a Zipf distribution experiment. However, with the static placement and partitioning strategy the system may be non-adaptive and thus may be less efficient when the load is, for instance, low.